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1. Workshop Objectives 


A workshop entitled “Outstanding Research Issues In Systematic Technology Prioritization for 
New Space Missions” was held April 21-22, 2004 in San Diego, California on behalf of NASA 
Program Managers Robert Pearce (Code R Division of Strategic Planning) and Doug Craig 
(currently in the Human and Robotic Technology Program of Code T). The purpose of this 
meeting was to explore the state-of-the-art in decision analysis in the context of being able to 
objectively allocate constrained technical resources to enable future space missions and optimize 
science return. 


The participants in this workshop are listed below: 


John. D. Azzolini 
Jacob Barhen 
David Bearden 
Doug Comstock 
Jason Derleth 
Mark Drummond 
Alberto Elfes 
Joseph Fragola 
Dave Beals 
Jalal Mapar 
Othar Hansson 


Goddard Space Flight Center 
Oak Ridge National Laboratory 
Aerospace Corporation 
NASA HQ Code BX 
Jet Propulsion Laboratory 
Ames Research Center 
Jet Propulsion Laboratory 
Science Applications Inc. 
Langley Research Center 
Science Applications Inc. 
Thinkbank, Inc. 


Louis Lollar 
Jon Neff 
Stephen Prusha 
Guillermo Rodriguez 
Paul Schenker 
Jeffrey Smith 
Raphael Some 
Mark Steiner 
Charles Weisbin 
Alan Wilhite 
Giulio Varsi 


Marshall Space Flight Center 
Aerospace Corporation 
Jet Propulsion Laboratory 
Jet Propulsion Laboratory 
Jet Propulsion Laboratory 
Jet Propulsion Laboratory 
Jet Propulsion Laboratory 
Goddard Space Flight Center 
Jet Propulsion Laboratory 
Georgia Institute of Tech. 
NASA HQ Code S 


2. Invited Talks 

Several invited speakers presented their approach and results of recent experience to provide 
background for the ensuing group discussions. 

The need for systematic technology assessment and prioritization was motivated in the talk 
entitled, “Strategic Investments Overview” by Doug Comstock, Director of Strategic 
Investments for NASA Code BX. Emphasis was on the demonstration of alignment of theme 
plans with the broader Agency Strategic Plan, and development of common analysis standards. 

Then, each of the two mornings was comprised of presentations from the following speakers: 

• “Estimating the Risk of Technology Development,” Alan Wilhite, Professor, Georgia 
Institute of Technology /National Institute of Aerospace. This talk discussed the 
characterization of risk through a matrix of probability and consequence. The probability 
was in turn, decomposed into probability of achieving technological maturity, and 
probability of achieving performance specifications, for a given resource allocation and 
schedule. An analytical hierarchical process is used to elicit data from experts. Specific 
case studies were used to illustrate these concepts. 

• “Technology Assessment of NASA Lidar Missions: A Pilot Study,” Mark Steiner, 
NASA Goddard Space Flight Center. This is a technology investment case study leading 
to a next generation LIDAR instrument. Science measurements needs were determined, 
and physics models developed which would enable mapping between technology 
performance and instrument performance. Future extensions were suggested in terms of 
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broadening the entire architeeture trade spaee and eombining available data/tools into a 
unified system. 

• “The Atlas Decision Support System,” Louis Lollar, NASA Marshall Space Flight 
Center. This talk discussed plans for the ATLAS system, intended as a single (high 
level) desk top tool which would integrate information concerning missions, 
architectures, technologies etc. with coverage across the full life cycle, and would 
recommend relative ranking of technological candidates. The system currently uses 
system mass (surrogate for cost) as the major discriminator. 

• “The Earth Science System Analysis Model,” Othar Hansson (Thinkbank, Inc.). This 
talk presented a 3-part investment model of technology change, impact assessment, and 
prioritization in the framework of an influence network for improved reliability of 
weather prediction. The example included 13 candidate technologies as they influence 

12 system characteristics (of the 13 x 12 = 156, only 1 8 are non-zero), with projected 
impact on 5 major system performance and cost metrics. An important consideration is 
that priorities depend on customer perspectives and there are often many different 
stakeholders (e.g. those interested in science, those interested in economics, those 
interested in safety etc.). 

• “Multi-Mission Strategic Technology Prioritization Study,” Charles Weisbin, Jet 

Propulsion Laboratory, California Institute of Technology. This is a comprehensive JPL 
study to date on technology assessment and prioritization. The START methodology 
described in section 1 demonstrated this approach can be used to assess a wide range of 
missions and technologies and is capable of inter-program trades. The study comprised 

13 missions and 167 technology performance parameters in 23 technology areas. 
Technology investment recommendations were provided at technology task and 
technology area level as a function of resources available. At any level of resource 
investment, the likelihood of missions being technologically enabled was also presented. 

The slides for these presentations are given in Appendix A. 

3. Group Discussions 

Each of the two afternoons was devoted to breakout sessions, addressing important questions and 
issues of current interest. Appendix B contains a detailed record of these discussions prepared by 
the breakout groups. Some of the more important highlights of these discussions are 
summarized below. 

Question 1: In prioritizing technology development for missions, how should the relative 

values of the missions be assessed and quantified? 

• Should mission (= flight project) value be assessed at all? Value is always assigned: 
current processes do this in a non-traceable, non-auditahle way. It has to he done, so that we 
can improve on today’s process. To do this, focus on functional objectives. The tool should 
allow for externally prescribed inputs about mission value. 

There will always be a difference between valuation theory and results versus a final 
assessment by the decision-maker. In making a final assessment, the decision-maker can 
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augment the evaluation results with other factors external to the analysis. Identifying the 
decision analysis process as a tool for mission and technology portfolio selection reduces 
political sensitivity about the relative position in the launch queue. 

• Who should do it? Can it be done? There is the problem of different stakeholders. Possible 
approaches are: (1) Code B assesses relative value of missions (they allocate resources to 
Enterprises). An example may be to consider the 18 theme areas and 3 mission areas, and 
given them each a high, medium, or low ranking; (2) Enterprises: Code B apportions 
resources as a block to Enterprises, Enterprises prioritize missions, with inputs from Science 
Groups and Project Managers; and (3) Executive Council, Joint Strategic Assessment 
Committee performs the prioritization. 

• How should it be done? There were many alternative suggestions offered. Stakeholders can 
assess mission values in a process not unlike that used to rank departments at various 
academic institutions. Project managers can be surveyed to provide input to this process. 
Another option is to count the strategic goals within the NASA Strategic Plan that are 
satisfied, and use this as a factor in assessing mission value. In another option, mission cost 
can be used as a surrogate for value, and relative prioritization can be expressed through 
budget deltas by theme from year to year. Yet another option is to assign value on the basis 
of classifying missions into those that enable entirely new scientific discoveries, and those 
that enhance scientific knowledge about phenomena that have been previously discovered. 
The NASA Strategic Plan should identify the “owner" of the prioritization process. 

Question 2: There are many architectural options to enable a mission, but at the early 

formulation stage, how might we best select among them, and perform a 
functional decomposition to determine quantified capability requirements? 

• It is possible to obtain mission capability requirements for missions that are at the early 
formulation stage. In many cases, particularly where there may be a vast spectrum of 
previous missions from which to draw data, requirements for new undefined missions can 
often be obtained by projected evolution. One can assume an evolution from the 
technological state of the art (technology push) and iterate between what the technology 
might be able to achieve, and the corresponding new mission requirements that can be 
satisfied. A functional decomposition is derived from mapping mission capability 
requirements to technology’ performance metrics. The functional decompositions from each 
new advanced concept study might be stored in a NASA database. Capability requirements 
for missions can be obtained to whatever level of detail may be available. Mapping relevant 
technologies to capability requirements can identify technology gaps, and these gaps can be 
used to derive performance metrics for technologies. The fulfillment of requirements can be 
evaluated by modeling and simulation or by analyzing the degree to which relevant figures of 
merit are satisfied. A relative value to various figures of merit may be assigned by 
parametric weighting of mission values and by conducting iterative sensitivity analysis. 
Don 't over-weigh optimizations but consider the level of precision; reserve some fraction for 
visionaries and spontaneous discoveries. Consider approaches from other sectors 
(government, non-NASA, public, etc.). 

• There are advantages and disadvantages of establishing requirements. “Requirements” 
are not ironclad, but have to be adaptive and negotiable. Requirements have to be coupled 
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with affordability' and serve as a basis for negotiation among mission and system designers 
and the related technology ’ developers. Requirements should ideally be expressed 
quantitatively. Requirements are different from specifications. Quantification of requirements 
can bring problems, but can also allow one to know when one is done. 

• Defining mission concepts involves working in a very large trade space. How do you 

search it? Search trade space hierarchically, keeping the number of options low at each level 
Delay decisions on final designs’. NASA tends to dive into a specific point design too early. 
A more extensive assessment of the trade space, keeping uncertainties and options open, 
allows a broader, more valuable set of technologies to be developed. On the other hand, there 
are huge costs associated with keeping options open. 

Question 3: How do we systematically acquire credible information, such as cost and 

performance estimates about technology development, which might seek to 
satisfy capability requirements. 

• Strive to make the data models and assumptions traceable and transparent. One of the 

key features in achieving data quality is to undertake an independent review of the data, by a 
team external to the data generation process. Workshops can be used to enhance credibility 
of the data collected. 

• Strive to obtain statistically significant samples in the data set. For high-risk or non- 
legacy technologies, the data should include estimates of uncertainty. In matching capability 
requirements to technology tasks, the data estimates should include as many valid viewpoints 
as possible to reduce the influence of inevitable uncertainties in individual data values. The 
larger the number of viewpoints represented in the data, the greater the robustness of the 
conclusions that can be drawn from it. 

• Strive to implement a data collection process that is sustainable. The POP process is a 
good programmatic vehicle to request data generation and to implement incentives for 
proper response to such requests. Iterations should be easier than the first bounce. The 
process for data collection should be continually reevaluated. Quarterly reviews of the 
information should be conducted with researchers, technology developers, and mission 
experts. 

Question 4: What is the best methodology to perform technical risk assessment, 

management and mitigation? Is the representation needed for risk 
management technologies fundamentally different to that needed for 
discipline-product technologies, such as sensing, manipulation, and thermal 
control? 

• The representation and assessment of risk estimation and software technologies should be 
made consistent with those of the discipline product technologies (e.g., sensing, 
manipulation, mobility’, etc.), in order to allow comparative analysis. It is important to have 
researchers state what kind of performance metrics they hope to impact; missions should 
provide goals. 
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• Risk manifests itself in terms of cost and schedule (as well as performance) and these impacts 
must be assessed in an integrated fashion. Software and hardware might be combined at a 
capability level as opposed to a discipline level. 

• State of the art can be characterized, but the whole ‘ecosystem’ of software should be looked 
at, not just an algorithm, for instance. 

Question 5: What are the criteria management will use to judge the results of a 

structured technology prioritization analysis? 

The analysis and its results have to support and defend the eventual decision to stakeholders 
such as OMB, GAO, and others. The analysis should be traceable, transparent, understandable, 
and presented in a concise way. The analysis should document explicitly the important issues, 
assumptions and approximations, and should identify major uncertainties and other problem 
areas. The analysis has to address what the decision-maker cares about, including metrics and 
alternative options. The analysis should have the objective of providing decision-support tools 
and should provide options instead of point-solutions. The results should be cast as trades 
between risk and cost or between benefit and cost. The analysis should result in preferred 
recommendations and justifications spanning the decision space, not just negatives and 
consequences. The analysis products should be digestible and tuned for interpretation at the 
appropriate level. 

4. Recommendations for Future Activities 

The meeting concluded with a discussion of potential future activities, which included: 

• Formulate and conduct a pilot application project, in partnership with a selected theme and 
program management representing mission, technology', and financial planning 
organizations. Increase the fidelity of the data and analysis, if necessary by initially 
narrowing the scope of mission and technology' options 

• Report on workshop results to the NASA multi-center System Analysis Consortia 

• Provide input to POP guidance next February (e.g. types of inputs required) 

• Provide additional organized opportunities for further technical discussion and exchange on 
such topics as risk assessment and decision analysis methods (e.g., partial completion of 
tasks, handling of reserves, etc.) 

• Investigate potential concurrent applications of technology prioritization methods to other 
government agencies (e.g.. Homeland Security). Address prototypical questions of potential 
benefit to others. 


5 



Appendix A: Slides of invited Talks 

• Doug Comstock 





Strategic Investments Overview 


Technology Prioritization Workshop 
Doug Comstock 

Director of Strategic Investments 
April 21, 2004 



Vision and Mission: 
Our New Starting Point 


SK^A 

- -# 

\ The NASA Vision 

Tn improve life here 
To extend life to there 
To find life beyond 

The NASA Mission 

To understand and protect our home planet, 
To explore the universe and search for life, 
To inspire the next generation cf explorers 

as only NASA can 
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Robust Strategy for Scientific Discovery 
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1 Vision 

All performance must be tied to the NASA Vision 

* r r.. 3=* 

I Mission 

One NASA: Many Themes support each of 3 NASA Missions 


1 Goal 

7 Goals tied to the Mission + 3 enabling Goals 


1 Objective 

What is to be accomplished; owned by a single Theme. 


1 Outcome 

An important multi-year step on a Theme's roadmap 

_ L_ _ 

Annual 

Performance 

Goal 

Indicates annual progress towards achieving outcomes. 
Tied to a Theme's budget investment. 
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Office of the Chief Financial Officer (Code B) 


CHIEF FINANCIAL 
OFFICER 

Gwendolyn Brown 
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Deputy Chief Financial Officer 
for Financial Management 
Jack Blair (Acting) 
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Steven Isakowltz 
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Director 
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Director 
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Strategic Investments Division 


Strategic Planning & Performance Branch 


Program Analysis Branch 


Performance and Results Team 

• Strategic Plan 

• Integrated Planning 

• Performance Measurement 

• Performance Reporting 

• Budget & Performance Integration 

• Strategic Management 

Systems Analysis Team 

• Standards for Systems Analysis 

• Analytical tool development 

• Support to Space Architect 

• Cross-Enterprise analysis 


Space Flight Team 

• Space Shuttle 

• Space Station 

• Space & Flight Support 

Science, Education & Aero Team 

• Space Science 

• Earth Sciece 

• Biological & Physical Research 

• Education 

• Aeronautics 

Exploration Team 

• Transportation 

• Human and Robotic Technology 
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Code BX Products 

Annual Budget Request - Integrated Budget and Performance Document 
(IBPD) 

• Code BX led the design, development and integration of the IBPD 

• Totally revamped Congressional justification - well received 
■ Page count less than half with more information than before 

• Integrates budget with performance, setting government-wide benchmark 
Performance and Accountability Report (PAR) 

• Code BX leads the formulation, integration, production of the PAR 

• Met aggressive OMB schedule 

• On schedule for meeting even more aggressive OMB schedule this year 

Strategic Plan 

• Code BX led the formulation, integration and production of the plan 

• High quality plan, seven months ahead of schedule 

Integrated Planning 

• Code BX developed and implemented the plan for integrated Agency planning 
in support of the Associate Deputy Administrator for Technical Programs 

• Integrated set of planning documents being produced for the first time, 
including Enterprise Strategies and Center Implementation Plans 

• A planning ‘community’ has been established with significantly improved 
communications 

• Working with other Agencies to share best practices 



Code BX Products 


* Budget Amendments and Supplemental Requests 

- Code BX leads/supports strategy, drafting, integration and advocacy 

- FY 2003 Budget Amendment 

* Approved by OMB, adopted by appropriators 

- FY 2004 Supplemental Request 

* Approved by OMB and now appropriated 

* Performance Plans 

- Pre-IBPD FY 2003 performance plan was re-mapped to new strategic 
framework for the Agency 

- FY 2004 performance plan revised to increase measurability of outcomes 
» Management Tool Development 

- Code BX working with IFM Program and Chief Engineer to establish 
requirements and implementation plans for Erasmus 
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Systems Analysis 

• The systems analysis community across the Agency is often called upon 
to assess investment strategies. 

- “How do we demonstrate alignment with the Agency Strategic Plan in a 
standard way?” 

- Wide range of analysis: ISTP, technology portfolios, cross Enterprise 
activities, spacecraft mission trades, etc.... 

• There are no “best practices” or common analysis standards to enable 
“apples to apples” comparisons of results. 

- Decision makers and analysts will both benefit from an open and transparent 
approach to performing and employing analysis products. 

- Have found that such standards are welcomed and encouraged. 

• Code BX is seeking to catalyze a systems analysis ‘community’ among 
existing organizations dispersed across the Agency. 

- Budget process is a consumer of a great deal of Agency systems analysis 
products. 

- Currently engaged in dialog with systems analysis and systems engineering 
groups around the Agency on developing standards and a community. 

- Collecting inventory of tools, approaches, and environments from around the 
Centers. 

- Will conduct workshops and develop standards this year. 

- Goal is improved communications and strengthened capabilities, leading to 
better investment decisions. 
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Summary 

Significant changes are underway 

• Integration among the vision and mission, strategic plan, 
budget, and performance planning and reporting 

- Closer linkage of our budget estimates with our strategic plan, 
performance measures and institutional needs 

- Systems analysis efforts to improve linkage for better decisions 

• Integrated budget and performance information in a single 
document, linked to strategic plan objectives through new 
budget structure arranged in “themes” 

- Ensures consistency among critical documents 

• Annual and long-term performance measures directly traceable 
through the strategic plan to the vision and mission 

- Clear accountability for results through themes 

• Defined agency goals requiring multiple enterprises and 
themes, with interdependencies and shared accountabilities 

- Reflects the One NASA philosophy 

These changes will help NASA to achieve our Vision and Mission 
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Dr. Alan W. Wilhite 




Estimating the Risk of 
Technology Development 


Dr. Alan W. Wilhite 

Langley Distinguished Professor/Systems Architectures and Analysis 
Georgia Institute of Technology/National Institute of Aerospace 


256 . 683.2897 


Center for Aerospace Systems Analysis (CASA) 


rv 


When do you do risk analysis ? 


Risk analysis and response planning must be 
done during the initial planning phase of the 
project. Ideally, risk analysis and response 
planning is done during the project proposal 
phase and revisited on a regular basis. 

"70% of a project's cost at completion is committed 
by the time the first 5% of the project's budget is 
actually spent. " 
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The Elements of Risk 

Risk is composed of TWO elements: 

1. ) The UNCERTAINTY (expressed as a probability (Pf) of 
achieving a project performance objective 

AND, 

2. ) The CONSEQUENCES (Cf) of a risk event 

Risk= Pf x Cf 

Caution is needed, of course in using this approach. It is necessary to 
be wary of multiplying 2 pieces of information together to produce a 
figure which may ,make an account's eyes light up but be of little 
practical value to a project manager. 

Risk Assessment Matrix 



Low Medium High 


Probability of Failure 
(1 - Probability of Success) 
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Characterization of Technology Risk 

(utilization for system development) 


■ Probability of failure to: 

- Reach maturity for system integration 
(programmatic failure) 

- And meet Technical Performance Measures 
goals (technical failure) 

■ Impact on overall system performance of 
failing to meet TPM goals 

Measures of 

Probability of Failure 

• The Probability of Failure is measured by the three measures used for 
programs or projects - cost, schedule, and performance. 


Performance (technical failure) 



Cost Schedule 

(programmatic failure) 
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Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

• Requirements, requirements flowdown, interface requirements, etc. 

• Schedule 

- Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

• Cost 

- Defined cost for all milestones 

- Costs include NASA and contractor 

• Management and technical team (experienced) 


NASA's TECHNOLOGY READINESS LEVEL 
(Scale for Tracking Risk Reduction) 

9 - Actual system "flight proven" on operational flight 

8 - Actual system completed and "flight qualified" through test and demonstration 
7 - System prototype demonstrated in flight 

6 - System/Subsystem (configuration) model or prototype demonstrated/validation 
in a relevant environment 

5 - Component (or breadboard) verification in a relevant environment 

4 - Component and/or breadboard test in a laboratory environment 

3 - Analytical & experimental critical function, or characteristic proof-of-concept, or 
completed design 

2 - Technology concept and/or application formulated (candidate selected) 

1 - Basic principles observed and reported 

( Technology Readiness Level of 6 is usually 
required for Development 
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NASA’s 

Technology Readiness Levels (Software) 


System Test, 
Launch & 
Operations 


System/Subsystem 

Development 


TRL 9 
TRL 8 
TRL 7 


Technology 

Demonstration 


Technology 

Development 


Research to 
Prove Feasibility 


Basic Technology 
Research 



TRL 9: Actual system “mission proven” through successful mission operations 

Thoroughly debugged software readily repeatable. Fully integrated with operational hardware/software 
systems. All documentation completed. Successful operational experience. Sustaining software 
engineering support in place Actual system fully demonstrated. 

TRL 8: Actual system completed and “mission qualified” through test and 
demonstration in an operational environment Thoroughly debugged software. Fully 
integrated with operational hardware and software systems. Most user documentation, training 
documentation, and maintenance documentation completed. All functionality tested in simulated and 
operational scenarios. V&V completed. 

TRL 7: Initial system demonstration in high-fidelity environment (parallel or 
shadow mode operation) Most functionality available for demonstration and test. Well integrated 
with operational hardware/software systems. Most software bugs removed. Limited documentation 
available. 

TRL 6: System/subsystem prototype validated in a relevant end-to-end 
environment Prototype implementations on full scale realistic problems. Partially integrated with 
existing hardware/software systems. Limited documentation available. Engineering feasibility fully 
demonstrated. 

TRL 5: Module and/or subsystem qualified in relevant environment Prototype 
implementations conform to target environment / interfaces. Experiments with realistic problems. 
Simulated interfaces to existing systems. 

TRL 4: Module and/or subsystem qualified in laboratory environment Standalone 
prototype implementations. Experiments with full scale problems or data sets. 

TRL 3: Analytical and experimental critical function and/or characteristic proof- 
of-concept Limited functionality implementations. Experiments with small representative data sets. 
Scientific feasibility fully demonstrated. 

TRL 2: Technology concept and/or application formulated Basic principles coded 
Experiments with synthetic data. Mostly applied research. 

TRL 1: Basic principles observed and reported Basic properties of algorithms, 
representations & concepts. Mathematical formulations. Mix of basic and applied research. 


Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

Research and Development Degree of Difficulty 

- TPM gap 

» Requirements, requirements flowdown, interface requirements, etc. 

• Schedule 

Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

High risk items, work around 
Exit criteria for every milestone 

• Cost 

Defined cost for all milestones 

- Costs include NASA and contractor 

% Management and technical team (experienced) 
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Research anc | Development 

Degree of Difficulty (RD 3 ) 

R&D 3 

I A very low degree of difficulty is anticipated in achieving research and 
development objectives for this technology. 

Probability of Success in “Normal” R&D Effort > 99% 

II A moderate degree of difficulty should be anticipated in achieving R&D 
objectives for this technology. 

Probability of Success in “Normal” R&D Effort > 90% 

III A high degree of difficulty anticipated in achieving R&D objectives for this 
technology. 

Probability of Success in “Normal" R&D Effort > 80% 

IV A very high degree of difficulty anticipated in achieving R&D objectives for this 
technology. 

Probability of Success in “Normal” R&D Effort > 50% 

V The degree of difficulty anticipated in achieving R&D objectives for this 
technology is so high that a fundamental breakthrough is required. 

Probability of Success in “Normal” R&D Effort > 20% 


Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

• Requirements, requirements flowdown, interface requirements, etc. 

• Schedule 

- Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

• Cost 

- Defined cost for all milestones 

- Costs include NASA and contractor 

• Management and technical team (experienced) 
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NASA Program Schedule Actuals 


MER 

Gemini - Manned 
Skylab Workshop - Manned 
Mars Global Surveyor 
Pathfinder 
Centaur-G' - Launch Vehicle 
Voyager - Unmanned 
Viking Lander - Planetary 
Magellan - Planetary 
Viking Orbiter - Unmanned 
Apollo LM - Manned 
S-IVB - Launch Vehicle 
Apollo CSM - Manned 
Mars Observer - Unmanned 
Skylab Airlock - Manned 
S-ll - Launch Vehicle 
External Tank 
Shuttle Orbiter - Manned 
Spacelab - Manned 



□ ADPto PDR 
■ PDR to CDR 

□ CDR to Launch 


0 20 40 60 80 100 120 140 

Calendar Months 


Life Cycle Milestones 


DOD 5000 ioc foc 





AOA 

ANALYSIS OF ALTERNATIVES 

IOC 

Initial Operational Capabiuty 

SAR 

System Acceptance Review 

▼ 

D0DI 5000.2 

ASOR 

Annual Systems Operations Review 

MCR 

Mission Concept Review 

SDR 

System Design Review 

COR 

Critical Design Review 

MDk 

Mission Design Review 

SFR 

System Functional Review 

▼ 

NASA 6105 

OR 

Decommissioning Review 

NAR 

Non -Advocate Review 

SR 

SAFETY REVIEW 

FCA 

Functional Conficuration Audit 

OAR 

Operational Acceptance Review 

SRR 

System Requirements Review 



FOC 

Fun Operational Capabiuty 

ORR 

Operational Requirements Review 

SVR 

System FArification Review 



FOR 

Formal Quaufication Review 

PC A 

Physical Conficuraion Audit 

1RR 

Test Readiness Review 



FRR 

Fucht Readiness Review 

PDR 

Preuminarv Design Review 





ICE 

Independent Cost Review 

PRR 

Production Readiness Review 
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Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

Requirements, requirements flowdown, interface requirements, 
etc. 

Schedule 

- Defined schedule showing maturity increasing/adequate analysis 
and testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

Cost 


- Defined cost for all milestones 

- Basis of costs (FTEs, facilities, hardware, etc.) 

• Management and technical team (experienced) 


Low NOx Combustor 
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Many cupons tested 
Feeds sector test prog 
Continues during sector test prog 
Used for sector design refinement 
Essentially complete by FY95 
GE/NASA 


Combines components for integrated evals 

3 configurations tested 

Primary feed to annular test program design 

Secondary feed to core combustor test program design 

Uses non EPM materials 

GE/NASA 


3 generation tests of progressively complex design 
Gen I tests and Gen II design from separate contract 
P&W test feed annular rig test program design 
NASA test feed core combustor test program 
Uses non EPM materials 
P&W/NASA 


Applies to RQL configuration 

P&W/NASA participation 

Feeds annular rig test program design 


Same as 1. 0.2.6 
P&W participation 


Added shape fidelity over rectangular evals 
Two test series of angle configuration 
Feed core combustor test program design 
GE 


Feed products to test programs as developed 
NASA 


Evaluation of rectangular sector configurations 
Primary feed to annular test program design 


Low NOx Combustor 


Feed products to test programs as developed 
NASA 


Feed products to test programs as developed 
Universities 


1 -Pager Work Schedule 
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Assessing Technology Risk Using AHR 
(Analytical Hierarchical Process) 

• The AHP is based on the hierarchical decomposition of the 
prioritization or forecasting criteria down to the level at 
which the decision or forecast alternatives can be pair- 
wise compared for relative strength against the criteria. 

• The pair-wise comparisons are made by the participating 
experts and translated onto a numerical ratio scale. 

• The AHP mathematical model then uses the input pair-wise 
comparisons data to compute priorities or forecast 
distributions as appropriate. 


Analytical Hierarchical Process 


Individual Assessment 


Metric Interval 

Most Likely 

Relative Likelihood 

20 to 25 Units 

o 

5% 

As likely as 
35 to 40 



25 to 30 

o 

25% 

As likely as 
35 to 40 



30 to 35 

o 

75% 

As likely as 
35 to 40 



35 to 40 

• 

100% 

Most likely 
interval 




45 to 50 

o 

10% 

As likely as 
35 to 40 





Integrated Group Assessment 
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Technology Risk Assessment - Phase 3 
Summary Of Airframe Risk Assessments 


TA 


TECHNOLOGY PROJECT 


COST 


SCHED 


TECH 


2 

2 

2 

2 

2 

2 


2 

2 

2 


2 


2 

2 


STRUCTURAL HEALTH MONITORING - NORTHROP GRUMMAN 
METALLIC CRYOTANK - BOEING 
CERAMIC MATRIX HOT STRUCTURES - MRD 
DURABLE ACREAGE CERAMIC TPS - BOEING 
DURABLE ACREAGE METALLIC TPS - OCEANEERINC. 

INTEGRATED AERO-THERMAL & STRUCTURAL THERMAL 
ANALYSIS - NASA 

STRUCTURAL & MATERIALS/TANK/TPS INTEGRATION - NASA 


STAGE SEP & ASCENT AERO-THERMODYNAMICS - NASA 


MATERIALS & ADVANCED MANUFACTURING: PERMEABILITY 
RESISTANCE - NASA 


LIGHTWEIGHT INFORMED MICRO-METEOROID RESISTANT 
TPS - NASA 

ULTRA HIGH TEMPERATURE SHARP EDGE TPS - EMC 


CERAMIC MATRIX COMPOSITE -SOUTHERN RESEARCH 



Technology Risk Assessment - Phase 3 
Structural Health Monitoring (Shm) 

TA-2 Airframe Northrop Grumman 

MAJOR RISKS 

o Cost - Cost of 8,000 sensors for full scale SHM could be very high, but is 
understood. 

Schedule - Critical schedule issue is availability of Composite Cryo-tank for testing. 
SHM starting at TRL 4 in 2002. No development issues affecting schedule. 

CD Technical 

> Reliability - Integration of 8,000 sensors into one reliable SHM is a risk 

> Testability - Availability of Full Scale Composite Cryo-tank for testing to achieve 
TRL 6 

CONTINGENCY PLAN SUGGESTION 

Use a subscale tank (18 to 20 ft diameter) to test SHM system 

NOTE: Only new or updated comments are contained in this report. Refer to Phase 2 
report for complete evaluation. No significant change in evaluation from Phase 2. 


Show Stopper - Lack of Funding for Composite Cryo-tank for 

Testing 

NOTICE: This information is technical data w ithin the definition of the International Traffic in Arms regulation (ITARl and/or Export Control Administration Regulations (EARl and is subject to the 
export control laws of the United States. Transfer of this data by any means to unauthorized persons, as defined by these laws, whether in the U. S. or abroad, w ithout an export license or other approval 
from the U. S. Department of State is expressly prohibited. 
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Structural Health Monitoring (Northrop Grumman) 

Development Schedule 



1: They should meet this goal based on present information. 


2: NGC is starting with the SHM technology at a TRL level of 4 in 2002. They have plans to develop a structural 
health monitoring system and integrate it into a full-scale composite cryotank and complete test in 2005 
timeframe. So the critical element of this is really having available a full-scale composite tank with this system 
integrated into it in 2005. That's the biggest concern because the funding level could get cut on the full-scale 
development of a composite tank that is in a separate technology development/funding under GEN2. So, there 
are no major issues with respect to developing the SHM system that NGC is proposing here. The issue is with 
respect to the availability of a full-scale composite cryotank in 2005/2006 which could face some serious 
funding issues given that GEN2 is probably not going to carry two tanks to TRL = 6 (metallic and composite). 


5: If funding is maintained for the duration of the project, it is probable that it will come in on schedule. 


7: There is a trade-off that should be made between the amount of health monitonng and robustness of 
design/analysis. As the vehicle is used for repeated flights some of the health monitoring sensors will become 
inoperable and others will produce data that has increasing errors. At some point a decision will need to be 
made relative to how many flights can be achieved before the health monitoring system itself must be inspected 
and checked out for adequate performance. The cost of maintaining the health monitoring system should be 
weighed against the cost of increasing the robustness of design thereby reducing the need for health 
monitoring. The reliability of the health monitonng system must consider the sensors, the data system and 
everything that is needed to transfer the data from the sensor to the data system. The lowest reliability part of 
the system may be the vehicle installed data transmission lines (quite a nest of lines) which must pass through 
the vehicle requiring compromises to be made in other disciplines of the vehicle design. 


Goal: 2006 years 


Technology Success Data 


Technology Area: Airframe Technologies 

Technology Development: Composite Cryotank 


Development Cost 
Development Schedule 

2 Weighted Programmatic Success: 

External Inspection Interval 
Flight Mission Life 
Internal Inspection Interval 
Leak Rate 
Operating Pressure 
Reliability 
Weight/Volume 


Combined Weighted Success: 



Assumption: The Low to High range contains 
100% of the possible v alues of the metric. 


Expected Value - Mean or 
average value of the 
estimated probability 
distribution. It is the value 
of the metric expected by 
the evaluators 


Expected Value Deviation - 
Deviation of the EV from the 
goal, calculated as follows: 

Absolute Value: EV - Goal 


Goal 

A minus sign in front of the 
calculated value indicates that 
the EV’ is worse than the goal. 


’ EV Deviation show by how much the EV misses the goal. It is omitted for certain metrics. 

Weighted Success is the average success probability of the metrics. 

3 Combined Weighted Success is average of technical and programmatic Weighted Success. 


0% - 20% 20%-50% 50%-100% 
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Launch Vehicle Propulsion Technoloav Selection 



Me&ized 


Advanced IVfeterials 


Chanter Ressire 


Ocntustion Bfid 


OF Ratio 


What is the your investment order? 
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Weighted Technology Impact Ranking 

(Quantitative assessment after tech portfolio selected and funded) 


Safety (45%) 

Loss of Crew 
Loss of Vehicle 
Loss of Mission 
Loss of Payload 
S/lb (35%) 

Launch Availability 
DDT&E - Average 
1st Unit Prod. Cos 
Annual Ops Cost ( 
Facilities Cost (10 
Technical (20%) 

Vehicle Empty Wei 
Vehicle GLOW 
Total Weighted Score 


in 

c 

CD 

E 

CD 


O 

O' 



Technologies 
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0.18 0.18 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.10 0.07 0.07 


0.19 0.11 0.13 0.09 


ESI 

ESI 

EE 
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■tin 
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■mn 

■mn 

ES 
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■mi 
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■mn 
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■mn 
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ES 
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■mn 
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■mn 
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ES 
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ESI 
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0.11 0.23 0.11 0.09 0.14 0.16 0.11 0.11 0.08 0.08 0.11 0.11 0.11 0.11 0.08 0.06 0.02 0.11 0.06 0.02 


0.39 0.27 0.34 0.50 



■mnmm 

■mn 

■mn 

■mn 

■mn 

■mn 

■mn 

■mi 

■mnm 

in mm 

■mn 



0.06 0.06 0.00 0.00 


0.41 0.38 0.36 0.36 0.34 0.30 0.30 0.30 0.25 0.26 0.22 0.27 0.13 0.09 


Impact Assessment 


High 




Low 



Comments on Investment Strategy 
and Impact Assessment Method 


• Very poor choice of technology portfolio (~two-thirds of 
technologies have low or negative impact) 

• Wrong requirements were developed 

• Systems analysis did not model the technologies 
correctly 





Impact on Requirements 
(weighted value functions) 







Technology Agency Impact Model 


Requirements Enterprise 

Flowdown Strategic 


Missions / 
Program 


Architecture 


Priority of missions within an Enterprise 


Percentage of total missions that architectures are utilized 


Percentage of proposed architectures that capability impacts 


Capability 


Technology Needs Technology 


Indexed technology impact on capabilities computed by systems 
analysis (not yet available for all Architectures) or by expert 
opinion 


Technology _ Capability . Architecture * Mission * Enterprise 
Impact Impact Impact Impact Impact 


Summary 

Technology Risk Assessment 

• Technology risk is based on the probability of technology 
development success versus the impact of the technology on 
the system 

• Technology development probability of failure is similar to any 
project. Should have defined WBS, requirements, schedule, 
cost, etc. 

• Expert opinion is used for assessment; AHP is one method to 
obtain and integrate the opinions. 

• Expert opinion or systems analysis can be used to define the 
impact of the technology on the system. 

• For total Agency impact, future enterprise missions need to be 
prioritized to assess technology global impact and risk. 
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• Mark Steiner 


Systematic Technology Planning - 
GSFC Perspective 

April 2 1 , 2004 


Mark Steiner 
Goddard Space Flight Center 
Greenbelt, MD 20771 

Introduction 

How do we integrate systematic technology 
investment planning into the process of 
architecting NASA's new space missions? 

• GSFC perspective based on: 

Exploration Initiative and current mission planning 
environment 

FY 2003 Lidar Technology Pilot Study w/ LaRC 
FY 2004 TAA study w/ JPL 

• Goddard’s vision as to what needs to be done next 
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Engineering and Technology 
Support Across Life Cycle 

Strategic technology investment analysis enhances ... 

Pre-formulation /Formulation 

Roadmap generation and review - Technology development and review 

Advanced concept development and review - Tracking and execution of roadmaps. 



_ _ r , . . advanced concepts, technologies, etc. 

Refinement of roadmaps, advanced „ r ... 

concepts, technologies, etc. ~ Requirements and Systems Analysis 

- Proposal development and review 


Cross Life Cycle Activities 

- Risk management 

- Project/ Program cross-coordination and cross-coupling 
Independent technical/ management review 

- Lessons Learned Identification & Feedback 



Implementation & Decommissioning Approval 

- Requirements management - Technology planning 

- Design and development of missions, - Approval review engineering and 

instruments, systems, technologies, etc. product support 

- Product and service delivery - Program/ Project plan support 

- Integration & test 

- Launch, early-orbit check-out 

- Operations & sustaining engineering 

- Technology Commercialization 

... sound decisions across mission and program life cycles. 
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Lidar Pilot Study: Charter from Code R 

Code R tasked GSFC and LaRC to perform a technology 
assessment study of Lidar missions with the following objectives: 

1 . Develop a process for assessing the system-level benefits of new 
technology investments to guide program investment decisions. 

2. Establish performance goals for evaluating the progress of technology 
development & risk relative to the state of the art. 

3. Identify high-payoff crosscutting technologies that are enabling for sets 
of future mission concepts with similar scientific objectives. 

GSFC and LaRC performed this Technology 
Assessment Analysis (TAA) pilot study 2003 

- Used system engineering approach to 
determine expected return on technology 
investments that could ultimately be used at 
the mission, enterprise, or agency level 

- Allowed specific technologies to be evaluated 
for their impact on life cycle cost 



Study Flow - 1 

Science inputs 

Captured science goals for aerosol Lidar - 

• Examined ESTIPS database to establish science 
objectives for next generation Lidar and found that more 
detailed information was needed. 

• Performed survey of aerosol-climate community and 
Lidar experts to fully populate domain of science 
measurement goals (e.g., detect aerosols and clouds and 
obtain their optical characteristics). 

Derived science measurement needs that drove the 
integrated instrument performance requirements (such as 
SNR for atmospheric area of interest). 
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Technology Development Risk 

Huge Potential Payoff 


Visionary 
Solutions 

1 

Low Technology 
Readiness Level 

Always a Trade-Off in Technology Investments 



High Technology 
Readiness Level 



Proven 

Technologies 



Low Risk 
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i echnology Development Modeling 


Technology 

Investments 


Technology 

Development 

Module 


Technology 

Performance 


System 

Performance 

Model 


f (TRL, Investment) 


Mission 

Enabled 


Technology Development Model 

(from starting TRL to TRL 6) maps 
development risk and investment 
plan (estimated schedule and 
budget) to technology 
performance over time. 


System Performance Model 

maps technology performance 
into system performance 


Link models and use them to trade 
off cost, development risk, and 
system performance to optimize 
technology investment plan. 


Systems Dynamic Modeling - 
Technology Development 
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The Study Methodology Enables 



to determine return on investment 


Combining lidar technology 
development modeling . . . 



and lidar performance modeling 



and provide best estimate as to which 
group of technologies would enable the 
mission, reduce cost, and be most 
likely to enhance overall value. 
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FY04 TAA Study 



Li dar Pilot Stinh/ FY03: 

r Develop an approach to 

maximize the value of NASA's 
technology investment. 

r Understand process of 
gathering information, 
developing models, and 
presenting results." 

r Develop a general approach for 
optimizing technology 
investments and applv to 
LIDAR measurements 


Expansion in FY04: 

r Partner with JPL to extend process to 
space architect's Design Reference 
Missions 


Work with other centers (LaRC, ARC) 
to broaden technology databases, share 
processes, share results 

Extend performance modeling to 
include instrument accommodations 
(spacecraft and ground system) 


Unified Agency-Wide Technology Assessment Framework 


Unified Technology Assessment 
Framework 


•Toolbox approach 
•Each tool is unique 
•Different views based on same 
data 

•Each tool optimizes over a 
specific dimension, depending 
on question being asked 
• Convergence results in Unified 
Process and helps V&V tools 


JPL Tool Set 
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Reference Missions & Grand Challenges 


Reference Missions 
(not listed in order of priority) 

Grand Challenges 

Orbital Aggregation and Space Infrastructure 
Systems (OASIS) 

Modular, Distributed Structures, Human Protection, Robotic 
Assembly 

Mars Surface Missions (e.g. Mars Science 
Laboratory; Astrobiology Field Lab; etc.) 

Long-Range Mobility on Ice; Deep Drilling; Automated 
Return Launch; Risk Mitigation (Pre-Phase A) 

Lunar Survey Study Mission 

Sensor Webs & Data Fusion: Lidar/Radar Instrument Systems; 
Multi-Spectral Scanner; Model-Driven Multi-Measurement- 
Validated Data Reduction 

Earth Biomass (surface, mid-canopy, and canopy 
heights. 

Lidar/Radar Instrument Systems; Multi-Spectral Scanner 

Sensor Webs & Data Fusion 

Model-Driven, Multi-Measurement- Validated, Data Reduction 

RASC - L2 Earth Observing Telescope 

Large deployable mirrors, membrane type shape control, 
formation flying 

Venus Surface Missions 

Extreme Environments (460C temp; 90 bar pressure; sulfuric 
acid clouds at 50 km) 

Generic Critical Design Review requirements 
derived from Pathfinder, Space Station or other 
recent mission 

Quantify mission-level impact of ECS technologies, such risk 
management and human organization, whose primary 
contribution is to the design process, and that arc not 
necessarily embodied within a hardware or software flight 
system 


NOTE: GSFC and JPL will share performance data on all reference missions. 


Studv Data Gatherin 



• Have developed a technology list in cooperation with JPL 

Shows who will gather technology information in which areas 

• Have common technology data gathering template, based 
heavily on Space Architect work 

• Common technology data template and sharing of this and 
the reference mission performance information will allow 
JPL and GSFC to run common data through both sets of 
tools and provide results for comparison 

• Analyze differences between tools, since view problem 
from different but complementary angles: 

JPL - good for matrixing many technologies across many mission 


GSFC - good for in-depth analysis of technology development 
within particular mission (performance parameter) set 
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Integration of Kisk into Technology 

Planning 



Guesswork/Gut Feel Replaced with Integrated System Analysis 
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Considerations for NASA 

Currently - 

• We conduct deterministic and probabilistic assessment of existing systems 
based on mission requirements 

- Probabilistic sensitivity analysis for point solutions (Shuttle, Station, ...) 

> system decision trees are often complex and may not capture everything 

Future - 

• Assessment of entire architecture trade space to include technology 
development risk, programmatic risk, operational risk (vehicle, etc.) and 
cost 

Effect of technology on system design/development/cost/schedule 

• Models to develop probability distribution of expected outcome 

Probability based Genome Model will integrate TRL to provide a powerful 
view into future mission strategies and architectures. 


Next Steps for NASA 

• Get all technology players to play together 

• Integrate processes and tools as makes 
sense to answer questions at the appropriate 
level 

• NASA Technology Assessment Technical 
Committee?? 

Unified Agency-Wide Technology 
Assessment Framework 
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• Louis Loilar 


“ATLAS” 

Advanced Technology Life-cycle 
Analysis System 

April 2004 

Louis F. Loilar 

Advanced Projects Office of the Flight Projects Directorate 
NASA/Marshall Space Flight Center 
Huntsville, AL 


John C. Mankins 

Deputy Director for Human and Robotic 
Technology 

Development Programs Division 
Office of Exploration Systems (Code T) 
Washington, DC 


Daniel A. O’Neil 

Advanced Projects Office of the Flight 
Projects Directorate 
NASA/Marshall Space Flight Center 
Huntsville, AL 


Contents 


• Overview 

• ATLAS Conceptual Diagram 

• ATLAS Architectural Overview 

• Notional Example 

• Summary 
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Overview 

• Making good decisions concerning research and development 
portfolios — and concerning the best systems concepts to pursue — 
as early as possible in the life cycle of advanced technologies is a 
key goal of R&D management 

• This goal depends upon the effective integration of information 
from a wide variety of sources as well as focused, high-level 
analyses intended to inform such decisions 

• The presentation provides a summary of the Advanced Technology 
Life-cycle Analysis System (ATLAS) methodology and tool kit... 

- ATLAS encompasses a wide range of methods and tools 

- A key foundation for ATLAS is the NASA-created Technology Readiness 
Level (TRL) systems 

- The toolkit is largely spreadsheet based (as of August 2003) 

• This product is being funded by the Human and Robotics 
Technology Program Office, Office of Exploration Systems, NASA 
Headquarters, Washington D.C. and is being integrated by Dan 
O’Neil of the Advanced Projects Office, NASA/MSFC, Huntsville, AL 


“ATLAS” Approach 

Advanced Technology Life-cycle Analysis System 
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Advanced Technology Life-cycle Analysis System (ATLAS) Model 

Architecture Overview 


Comm 

Mission 

WPT 

Test 

Systems 

Ops 

Facilities 


Science 

Missions 


Human 

Exploration 

Missions 


| Commercial 
Space 
Markets 


Users... 


ATLAS “coded" 
Interface 


ATLAS “virtual" 
Interface 
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Scenario Integration 
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Notional Example Analysis 

Lunar Rover to Collect Ice from the Lunar Craters 

• Notional Scenario 

- Launch elements to LEO for construction 

- LEO to Lunar Orbit 

- Base system/Rover to “Edge of Crater” 

- Rover descends into the crater to retrieve some ice 

- Rover brings the ice back to the base unit 

• Analyst chooses(with help from ATLAS) 

- Launch Vehicle 

- LEO Base Configuration 

- Orbital Transfer Vehicle 

- Base Vehicle 

- Lunar Rover 

• Output Data from ATLAS 

- Mass statement(s) for each subsystem and/or 18 subsystems 

- DDT &E (6 year cycle) 

- Cost for each system and/or 18 subsystems 

- Theoretical first unit cost 

- Life cycle costs 

- Views of the intermediate steps of the process 
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Summary 

• A central challenge in the management of innovation 
lies in making good decisions in the absence of 
complete information 

- The conundrum is that the earliest decisions have the greatest 
affect on project outcomes, and yet they must be made at the time 
when there is the least detailed information available 

• The ATLAS modeling system is being developed to 
contribute to the resolution of this challenge 

- By providing a single (high-level), desk-top tool that integrates 
information on, and analytical relationships among various missions, 
architectures, systems, technologies and associated metrics, and 
costs 

• Although considerable work remains, it appears likely 
that ATLAS will begin operations — and to make 
meaningful contributions to Agency decisions — during 
FY 2004 
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• Othar Hansson 


The CICT Earth Science 
Systems Analysis Model 

M COMPUTING 

INFORMATION & 

COMMUNICATIONS 

TECHNOLOGY 

Barney Pell, Joe Coughlan, 
Bryan Biegel, Ken Stevens, 
Othar Hansson, Jordan Hayes 

NASA Ames Research Center 
& Thinkbank, Inc. 

April 2004 


The ESS A Team 


• Task leads: 

Barney Pell (Lead), Bryan Biegel (Co-lead), 
Joe Coughlan (Science Lead), 

Walt Brooks (Science Co-Lead) 

• Subcontractor: 

Othar Hansson & Jordan Hayes, Thinkbank 

• ARC team: 

Ken Stevens, Peter Cheeseman, Chris Henze, 
Samson Cheung, et al. 


46 







Enough About Me 

• Research collaborations with NASA Ames since 1989 
(heuristic search, data-mining, planning/scheduling). 

• PhD (Computer Science), Berkeley. 

Using decision analysis techniques for search control 
decisions in science planning/scheduling systems. 

• Thinkbank: 

custom software development, 

software architecture consulting, 

technology due-diligence for investors. 


Agenda 

CICT Systems Analysis 

Our modeling approach 

- a 3-part schematic investment model of 
technology change, impact assessment and 
prioritization 

A whirlwind tour of our model 
Lessons learned 
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Systems Analysis in CICT 

• Demonstrate "systematic and thorough investment decision 
process" to HQ, OMB and Congressional Decision Makers 

• Increase awareness and substantiate CICT's impact to 
missions. Road map CICT projects to missions and 
measurement systems 

• 4 teams in FY03: 

- 2 pilot studies (Earth Science [me]; Space Science [Weisbin]): 
explore models for ROI of IT. 

- TEAM: map from NASA Strategic Plan to IT capability 
requirement; technology impact assessment 

- Systems Analysis Tools (COTS/GOTS) 


Earth Science Pilot Study 

How do we characterize and quantify a 
science process? 

Can we build a model of how CICT 
technology investments impact ROI in a 
NASA science process? 

What modeling approach is suitable for 
making such analyses understandable and 
repeatable? 
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Current State 


What have we learned? (FY03) 

• Decision analysis modeling techniques can be 
applied to systems analysis of CICT project areas. 
Built model of weather-prediction data pipeline. 

What don't we know? (FY04) 

• How much time/expense needed 
to build a full model 

• How such a full model fits into a real 
NASA program context 

(CDS: Collaborative Decision Systems) 


Pilot Study Focus 

• Criteria for science process to study 

- Important to a major customer base, 

- Significantly drives technology investments 

- Generalizes to a class of related processes 

- Amenable to quantitative analysis. 

• 2010 Weather Prediction process 

- Critical Earth Science process with relevance not only to 
NASA scientists but to the nation at large. 

- Stretch goals require technology breakthroughs. 

- Strong technology driver for other science problems 

- Starting point: analyses from ESE 

computational technology requirements workshop (4/02) 


49 





Pilot Study Accomplishments 

• Identified modeling formalism (influence 
diagrams) 

- Clear semantics accessible to both ES & CICT experts 

- Tools exist for sensitivity analysis, decision-making, 
etc. 

We chose Analytica as our modeling tool. 

- Successfully transferred/applied to Space Science pilot 
study as well. 

• Built a model with an understandable, simple 
structure (after much research and many 
iterations). 

• Demonstrated the kinds of analyses made 

possible by the model 


Agenda 

CICT Systems Analysis 

Our modeling approach 

- a 3-part schematic investment model of 
technology change, impact assessment 
and prioritization 

A whirlwind tour of our model 

Lessons learned 
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Methodology: Decision Model 


Overall 
System Value 


Ql: Which technology investments should I make? 

Q2: How does each technology investment improve 
overall system/mission value (including cost 
considerations)? Choose investments with highest 
value. 


Filling in the Decision Model 

System 

Performance & 


Technology 

Investments 


Model 

System value is a function of a set of metrics (accuracy, 
fidelity, cost, etc.). We can model the priority among 
the metrics independent of the technologies used. 

Technology investments have value in that they improve 
these metrics. 
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System 

Characteristics 


System 

Performance & 
Cost Metrics 


Overall 
System Value 


System- 

Assessment 

Model 


System 

Priorities 

Model 


The metrics can be modeled in terms of abstract system 
characteristics (data volume, algorithm accuracy, 
processing speed, model fidelity, ...). 


Filling in the Decision Model 



System 

Performance & 
Cost Metrics 


Overall 
System Value 


Technology 

Investments 


System 


Change 


Model 


System- 

Assessment 

Model 


System 

Priorities 

Model 


Technology investments, together with some mission- 
specific parameters, influence the system characteristics. 
A technology investment (such as data visualization 
research) has value in that it improves system 
characteristics (such as model fidelity). 











Methodology: Influence Diagrams 



We've sketched an "influence diagram" model of the 
decision. 


Q: What tech, investments maximize expected overall system value? 

Q: Value of model refinement: How sensitive to assumption A? 

Q: Value of information: what if we knew that project P would succeed? 
Q: Value of control: what if we could reduce risk of project P failing? 


Influence Diagram Details 



Influence diagram tools (such as Analytica) allow you to specify and 
evaluate these models. Diagram structure and decision analysis 
techniques speed specification of required parameters. 

"What-if" and optimization questions reduce to the problem of 
computing functions of conditional prob. distributions: 

"best" technology investment is: 

argmax [E(Overall System Value | Technology Investments)] 
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Agenda 

CICT Systems Analysis 

Our modeling approach 

- a 3-part schematic investment model of 
technology change, impact assessment and 
prioritization 

A whirlwind tour of our model 
Lessons learned 

The ESSA Model 


5 System 
Performance & 
Cost Metrics 



Model 


Our set of 5 metrics include: 

development cost, operations cost, accuracy, model fidelity, etc. 





The ESSA Model 


1 2 System 
Characteristics 


5 System 
Performance & 
Cost Metrics 


Overall 
System Value 


System- 

Assessment 

Model 


System 

Priorities 

Model 


Our 12 System Characteristics include: 

observation density, assimilation efficiency, cpu efficiency, etc. 


The ESSA Model 
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Diving Down into the Model 



System-Assessment Model: the most stable part of the model, 
owned/designed by a customer domain expert who understands the 
behavior of the system/mission being analyzed. 

System-Assessment model computes System Metrics from System 
Characteristics 


System-Assessment Model 



system 
characteristics 


) 


Here is the model of how system 
characteristics drive system 
system metrics (cost & other; 
utility attributes). We have tried 
to break the model into modules' ' 
corresponding to the processes in 
the underlying dataflow Each 
process (tan rounded rects) has 
several outputs (blue ovals) that 
are in turn inputs to subsequent 
processes These intermediate 
values decouple the models. of the 
individual processes 


System performance and cost metrics 
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Example System Characteristics 


Assimilation efficiency 
CPU efficiency 
Data efficiency 

Ensemble efficiency 
Model framework 
Observation density 


0-1 scale: how much information is retained 
despite approximations in data assimilation? 

>0 : percentage speedup in CPUs due to R&D 
investments 

0-1 scale: how much information is present in each 
bit of data selected? 

0-1 scale: how much improvement in forecast skill 
do we get from using ensemble algorithms? 

0-1 scale: how much fidelity is present in our 
models? 

0-1 scale: how many of the available observations 
do we make? 


Postprocessing 

effectiveness 

Simulation efficiency 


0-1 scale: how much improvement in forecast skill 
do we get from using post-processing? 

> 0: percentage speedups in simulation due to 
R&D investments 


Instantiating the Model 



System-Change Model: owned/designed by a program manager who 
understands the feasibility and impact of different research areas. 

System-Change model computes System Characteristics from the set 
of Technology Investments chosen (and system/mission config 
parameters) 
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System-Change Model 

• "Impact matrix" quantifies the changes to system 
characteristics that will occur if individual research 
projects succeed. 

• "Cost matrix" quantifies cost breakdown for each 
research area. 

• Portfolio of research areas determines what 
impacts will be felt. 

• (In an extended model, cost and impact could vary 
over time.) 


System-Change: Research Areas 

• Data -efficient simulations (same data size) 

choose a more informative set of observations to improve forecast skill at 
the same computational cost 

• Data -efficient simulations (less data) 

reduce number of observations (and reduce computational cost) w/o 
reducing forecast skill 

• Targeted Observing 

ditto, but also gather more targeted observations based on ensemble 
accuracy estimates (e.g., the SensorWeb concept) 

• Adaptive grid methods 

reduce number of grid points by using regional forecast as boundary 
conditions 

• Improvements in ensemble methods 

reduce number of ensembles needed to get similar accuracy estimates 
(e.g., through use of particle filter technology) 

• Data-mining of model outputs 

increased skill from same model output via data analysis & visualization 
(intelligent data understanding) 
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System-Change: Research Areas 

• Modeling tools 

ESMF and other initiatives to make modeling efforts more 
productive 

• System Management/Tuning tools 

Auto or Semi-Automatic Parallelization tools, Benchmarking, 
Cluster management, etc. 

• Instrument models 

tools for creating more accurate instrument models. 

• Launch new data source 

collect additional types of observation data by launching a new 
instrument. 

• Launch replacement data source 

collect a new type of observation data, but keep the total amount 
of data processed the same. 

• Higher resolution models 

develop higher resolution models and move to higher resolution 
simulation 


Research Area Impact 

Impact matrix has a value for each pair (13 research areas x 12 
system characteristics): 156 possible, but only 18 are nonzero. 

Impact can be positive or negative: 

Impact(targeted observing, observation density) = low neg. 

Impact(launch new data source, observation density) = low 

Some more examples: 

Impact(targeted observing, targeting efficiency) = low 
Impact(system mgmt/tuning, cpu efficiency) = low 
Impact(adaptive grid, simulation efficiency) = medium 
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impact Matrix 


data-efficient simulations 

(same data size) 

data-efficient simulations 

(less data) 

targeted observing 

adaptive grid method s 
improved ensemble 

methods 

data-mining of model 

outputs 

modeling tools 



system mgmtytuning 


launch new data source 


launch replacement data 
source 


instrument mode Is 

higher resolution models 



Qualitative -> Quantitative 


Impact is parameterized qualitatively (lo, med, hi). This 
qualitative scale is then quantified inside the model. 

Each of the parameters has a different interpretation 
under the four scenarios (pessimistic, consensus, 
optimistic, ideal). This allows us to compare in a best- 
case vs. worst-case manner. 



pess. 

cons. 

optim. ideal 

Lo 

.05 

.1 

.15 

1.0 

Med 

.2 

.3 

.4 

1.0 

Hi 

.3 

.5 

.7 

1.0 








Instantiating the Model 



12 System 

I 

I Characteristics 


5 System 
Performance & 
Cost Metrics 


Overall 
System Value 


System- 

Change 

Model 


System- 

Assessment 

Model 


System 

Priorities 

Model 


System Priorities Model: designed/owned by program 
manager cognizant of NASA priorities 

System Priorities Model computes overall System Value 
given th e System Metrics. 

System Priorities Model 


understanding 


Cost 

summary 


basic/applied 

charter 


\ 


value to 
scientist 
stakeholder 


value to 
coast-dwelling 
stakeholder 


ove rail 

system value , 


net 

improvement 
, in value 


increase in overall 


are we concerned 
with basic research 
or applied research? 


responsiveness 
i to charter 


value from 
NASA 
perspective 


The "utility model" index 
specifies whose priorities 
(which utility function) should 
be used in the analysis 


baseline 

calculations 


The baseline is defined as the 
system value if no additional 
tech investments are made 


system value as- 
compared to baseline 





Remember: results (evaluations, ROI, etc.) 
must be understood as a function of the inputs used 

to calculate the results: 

/■(model, assumptions, priorities) 


Priorities depend on perspective: 
we model basic (science value only) 
versus applied (economic value only) 
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nat improvement in valua I net improvement in value 


Evaluatina Research Areas 


Result net improvement in value 

IE2 

IDS | 

I EH 


Mid Value of net improvement in value 

utility model O [ MASA j E3 £3 

optimism over research outcomes O consensus ] 13 d 
proposal 2 ^ [__ none _ 

| proposal 1 ▼] l~ T otals 

™ basic/applied charter 


d^i'efflciert.inSArtio^t^iWra 

HaeaBmaH — ■ 

improved ensemblemett«odsl_. __ 

rfrfiJlillinnrf m nrtel m tmamtaHMm 

IMBHBWIIItM 

launch new data source 
launch replacement data source 
instrument models 
hioher resolution models 


BSZ> 


0 

806.9M 
806 9M 
982. BM 
3.577M ( 

1 S36G T 
2 56G " 
108. 1M 
3.577M 
1 .S28G 
-20 
159.2M 
0 


IZI@® 




▼ |t> r Totals 


Basic : launch new data source (35M) & targeted observing (22M) 
Applied : data-mining (2.5B) & improved ensemble methods (1.5B) 

Evaluatina Research Areas 


Result net improvement in value 


Mid Value of net improvement in value 
basic /applied charter Q 

utility model O 

optimism over research outcomes O 
proposal 2 O 


X A«is: | proposal 1 



none data-efficien. data-effici... targeted adaptive gri . .improved e. data-minir.. 

proposal 1 


modeling.. system mg... launch new... Isonchre... instrument higher re... 



none data-eff cien data-effic targeted adapt ve gri improved e. dataminn . modeling system mg launch tew lauichre. instrument.. higher re 

proposal 1 
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Sensitivity Analysis 

Sensitivity to "optimism" variable: two research areas have vastly higher 
potential impact under ideal assumptions. Pessimistic view of data- 
mining exceeds optimistic assessment of other areas. 



nor* data-tffic. data-tffici targeted idaprtiv* improva dtfa-mi modeling system launch launch r«.. instruma ... higher r«. 

proposal 2 


Kay optimism ovtr research outcomes 
pessimistic 
■■ consensus 
r—* optimistic 

r~— i ideal 


Synergy Between Research Areas 

We can look for synergies by finding pairs of research 
areas with much higher value than the two areas 
individually... 

Under the applied research focus: 

Biggest synergies 

Launch new data source ($1.5B) 

+ targeted observing ($1B) 
yields a synergy of $700MM 

Launch new data source ($1.5B) 

+ data-efficient simulations ($800MM) yields a 
synergy of $400MM 
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Understanding the Model 


( system 1 

characteristics \ 


/"data to asssimilate: 
bytes/interval 



r 


data selection 


process 
ch a ra cte risti cs 





v^AC > _ 7 ) 


BLUE OVALS summarize 
the way that system changes 
flow through the assessment 
model. We can diagnose our 
assumptions by analyzing 
how these variables vary as 
we vary research area. 


System performance and cost metrics 


Agenda 

CICT Systems Analysis 

Our modeling approach 

- a 3-part schematic investment model 
of technology change, impact 
assessment and prioritization 

A whirlwind tour of our model 

Lessons learned 
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Modeling lessons learned... 

Model and modeling technology should be: 

• understandable and easy to use 

and should support: 

• varying levels of detail (qualitative->quantitative) 

• varying scope 

(cross-cutting value as well as mission-specific value) 

• development of models by distributed stakeholders 

• multiple uses / answer multiple questions 

• varying assumptions/priorities 

• communication/debate/collaboration 


Lessons learned. .. 


• Model preferences of different stakeholders 
explicitly 

• Allow for easy variation in assumptions ("what if 
our model is wrong? ...our estimates overly 
optimistic?") 

• Compare impact of each technology to a no- 
investment baseline 

• Make models modular and decoupled: 
technology investments -> 

system characteristics -> 
performance metrics -> 

"return" or "mission value" 

(three arrows =- three submodels) 
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End of workshop talk... 

Full report is available at 

http: //support, thinkbank. com/essa-fmal 
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• Chuck Weisbin 


Multi-Mission Strategic 
Technology Prioritization Study 

C. R. Weisbin, G. Rodriguez, A. Elfes, J. Derleth, 

J.H. Smith, R. Manvi, B. Kennedy, and K. Shelton 

"Systematic Technology Prioritization For New Space Missions" 

Humphrey's Half Moon Inn, San Diego, CA 


Jet Propulsion Laboratory 
California Institute of Technology 
April 22, 2004 
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Study Staff & Roles 

>JPL 

■ J. Derleth, Mission & Technology Portfolio Optimization 

■ A. Elfes, ECS Data & Analysis 

■ B. Kennedy, ECT Data & Analysis 

■ R. Manvi, Tech Life Cycle & Risk Management Model 

■ K. Shelton, Mission & Technology Data Base 

■ J. H. Smith, Integrated Risk Analysis 

■ G. Rodriguez, System Analysis 

> GSFC staff (M. Steiner, J. Azzolini, J. Mapar, C. 
Stromgren) 


Study Objectives 

• Perform a pilot study of sufficient breadth which 
demonstrates in an auditable fashion how advanced space 
technology development can best impact future NASA 
missions 

- Include wide spectrum of missions & technologies 

- Can add new missions & technologies easily 

- Optimize technology portfolios 

- Lead to rapidly prototyped example 

• Show an approach to deal effectively with inter-program 
analysis trades 

• Explore the limits of these approaches and tools in terms of 
what can be realistically achieved (scope, detail, schedule, 
etc.) 
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Technology Portfolio Optimization Approach 

• Collect performance data for many individual 
technologies; each data input is viewed as a statistical 
sample representing an expert assessment 

• Group the technological data into a tree-like 
hierarchical model to predict “integrated” system, 
mission, and multi-mission impact of individual 
technologies 

i Search computationally for technology portfolios with 
optimal science return, risk and cost impact 

• Investigate sensitivity of the optimal portfolio to 
changes in available budget levels 


Major Study Challenges 

• Reference Missions : assess mission value; characterize capability 
requirements 

• Technology Projections : characterize performance; manage widely 
dispersed and non-uniform data 

• Uncertainty : incorporate & manage widespread uncertainty 

• ROl Measures : formulate suitable value function for portfolio 
analysis 

• Lavers of Abstraction : choose and maintain appropriate level of 
analytical abstraction 

• Technological Boundaries : boundaries of technology domains not 
clearly marked 

• Many Scales : large differences in cost and performance scales for 
different technologies 

• Performance Parameters : not fully understood for some technologies 
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Implementation Approach 

• Iterative in three phases (keep eye on big picture early, and 
continuously) 

- Phase 1 minimalist multi-mission set; ECT/ECS technologies 

- Phase 2 more extensive set of missions & technologies (June 04) 

- Phase 3 completion of full study (December 04) 

• Maintain high degree of connectivity 

- Space Architect 

- Revolutionary Mission Concepts 

- Advanced Space Technology Programs 

- Enterprises 

- Centers 

- Etc. 
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Reference Missions & Major Challenges 

(Minimalist Mission Set for PHASE I) 


Reference Mission Classes 
(not listed in order of priority) 

Major Challenges 

Earth's Moon: Orbital Aaareeation and SDace 
Infrastructure Systems (OASIS); Lunar Remote 
Survey; Lunar Surface Missions; etc. 

Deep Space Robotic Rendezvous & Docking; Long Term 
Cryogenic Fuel Storage in Space (>2 years); Long Life Ion 
Engines(>15 K-hours) 

Mars Surface: (e.a. Mars Science Laboratory: 
Astrobiology Field Lab; Mars Sample Return; etc.) 

Long-Range. Long-Life Mobility ( 10's of kilometers, >600 
sols); Substantive Sample Collection and Return (>lkg, 
0<depth< 1 00m subsurface) 

Earth Observation: Biomass 

Lidar/Radar Instrument Systems; Multi-Spectral Scanner; 
Sensor Webs & Data Fusion 

Outer Solar System: Titan Surface: F.urona Lander 

Extreme Environments; Sub-Surface Ice Mobility 

Inner Solar System: Venus surface: comet samnle 
return 

Extreme Environments (460C temp; 90 bar pressure; 
sulfuric acid clouds at 50 km) 


> Technologies to be evaluated will include: 

■ Technological products in several discipline fields (aimed at operational flight 
system implementation (e.q. advanced materials, structures, etc.) 

■ Risk assessment tools and infrastructure to allow for risk quantification, and risk 
mitigation during an entire mission life-cycle, but that do not necessarily appear in 
the flight system implementation (e.q. risk management methods) 


Enabling Technologies for Which 
Data Has Been Collected to Date 

• Extreme Temp & Pressure Components, Thermal Control, 
Pressure-Vessel-Encapsulated Electronics (Venus) 

• Electric & Chemical Propulsion; Reaction Control; 
Multifunction Structures; Fuel Storage & Control; Syntactic 
Foams, Formation Flying (OASIS) 

• Entry Descent & Landing; Surface, Aerial, Subsurface 
Mobility; Manipulation, Drilling, Sampling (Mars, Titan, 
Comet, Lunar Surface) 

• In-Space Inspection, Maintenance, Assembly (OASIS, Large 
Observatory Platform, Gateway, Space Solar Power) 

• Risk Methods, Tools and Workstation; Mishap Anomaly Data 
Base; Complex Systems Research; Risk Characterization & 
Visualization; etc. (All Reference Missions) 
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Enabling Technology Areas 


(for which data has been collected to date) 


Enabling Technology Areas 

Missions 

Electric & Chemical Propulsion; Reaction Control; Multifunction 
Structures; Fuel Storage & Control; Syntactic Foams, Formation Flying; 
In-Space Robotic Inspection, Maintenance, Assembly 

OASIS 

Entry Descent & handing; Surface, Aerial, Subsurface Mobility; 
Manipulation, Drilling, Sampling 

Mars, Earth’s 
Moon, Titan, 
Comet 

Risk Methods, Tools & Workstation; Mishap Anomaly Data Base; 
Complex Systems Research; Risk Characterization & Visualization; etc. 

All 

Extreme Temp & Pressure Components, Thermal Control, Pressure- 
Vessel-Encapsulated Electronics 

Venus, Titan, 
Europa | 


Technology Areas are Decomposed into Many 
Sub-Areas & Performance Parameters 


A Few Typical 
Technology 
Areas 

A Few Typical 
Technology 
Sub-Areas 

A Few Typical 
Performance 
Parameters 

Multi-Function Structures 

Modular. Distributed Structures. 
Deployable Staictures, etc. 

Contract/Extend (cm). Power per 
Mass (W/kg), etc. 

Fuel Storage & Control 

On Orbit Cryrogenic Fuel Transfer. 
Tank Pressure Control. Fuel Storage, 
etc. 

Flow Rate (kg/min). Pressure 
(kPa). Time (yrs). etc. 

Subsurface Ice Mobility 

Range, Radiation Dose. Payload 
Capacity, Ambient Pressure, etc. 

Distance (km, mRads), Mass 
(kg). Pressure (atm), etc. 

Extreme Temperature & Pressure 
Components 

High Temperature Electronics. 
Permanent Magnets, Energy Storage, 
etc. 

Temperature (Celsius), Pressure 
(Bars). Energy Density (Whr/1) 
etc. 

Risk Methods, Tools & 
Workstation 

Model Based Risk Analysis, Mission 
Risk Profiling Capability, etc. 

Accessibility, applicability to 
multiple mission phases, risk 
mitigation coverage 


This is an early draft for April 15 th , 2004. Please do not distribute. 
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Mission & Technology Data Base 


Mission & Technology Data Base 


-- Current Size Summary -- 

Size of Mission & Technology Capability Data Base (as of April 15, 
2004) 

- 13 missions covering wide spectrum of NASA strategic plans 

- 23 technology areas (structures, energetics, extreme environments, surface 
mobility, etc.) 

- 86 technology sub-areas (batteries, payload capacity, thermal control, etc.) 

- 167 technological performance parameters (power density, operating 
temperature, etc.) 

Remarks About Data Base 

- Current data set is more detailed in some areas than in others 

- More technologies & detail will be collected in subsequent phases 
Our analysis methods can handle data sets with non-uniform detail 


This is an early draft for April 15th. 2004 Please do not distribute 
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Risk Related Requirements 


(from Point of View of a Project Manager) 

Risk Management Must: 

- Delineate major risks: Technical, Human, Organizational, 
Budgetary, and Schedules ;estimate and rank risk levels 

- Provide ways to visualize risk elements, time profile, and 
mitigation strategies 

- Assure that the systems and trade analysis includes cost, 
performance, and risk 

- Provide auditable benefit/cost of implementing begin-to-end risk 
mitigation strategies 


Connecting Risk Technologies 
to Requirements 


Requirements: i 

□ Delineate major risks: Technical, 
Human, Organizational, Budgetary, 
and Schedules; estimate and rank risk 
levels 

□ Provide ways to visualize risk 
elements, time profile, and mitigation 
strategies 

□ Assure that a substantial portion of 
the design space is explored 
including cost, performance, and risk 

□ Provide auditable benefit/cost of §5 

implementing end to end risk -2 

mitigation strategies c m 
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ECS: Engineering of Complex Systems 

> SRRM: System Reasoning and Risk Management 

• KESS: Knowledge Engineering for Safety and 
Success 

• RSO: Resilient Systems and Operations 
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System Reasoning and Risk Management 
(SRRM) Project Executive Summary 


Goals 


Objectives 


Advance scientific and engineering 
understanding of system risk, 
complexity, and failure. 


i 

— - 1 



Develop processes & tools to identify, 
characterize, mitigate, trade, and track 
full lifecycle mission risks. 



Risks not well 
understood or well 

Challenges characterized, 

especially in early 
design phases 


Risk not an Data and interactions integration of tools & 

inherent resource in complex systems data of differing detail, 

in design tradeoffs are difficult to model context, and pedigree 

and visualize for variety of decision - 

makers 


Analyze & model 
events and 

Approach interactions which 
have lead to system 
mishaps and failures 


Develop capability to 
fully characterize and 
model risk signatures 
early and consistently 


Mature & improve 
fidelity of subsystem 
models to capture 
failure modes and 
consequences 


Broaden the design 
space by fully 
integrating models 
and demonstrating 
the utility of risk as 
a tradable resource 


Technoloqv Accessibility of 

_ historical risk 

Performance event data 
Attributes 


Potential to understand 
and reduce design 
risks and optimize 
resources to 
retire risks 


Risk model 
enhancement 
(potential for better 
model credibility) 


End-to-end risk 
integration for 
breadth of domain 


Degree of 
Alignment 
(Effectiveness 
in percent) 


Attribute Definitions 


Accessibility of 
risk data 


Potential to 
reduce design 
risks 


Risk model 
enhancement 


— z — 

x 


— X 

x_ 


X 

x__ 


Easy to use DB spans multiple mission/projects with risk events categorized 
for search. 

DB may be limited to specific category or series of missions. 

Supporting data/verifications are anecdotal (narrative) format without 
categories of risk events for easy search. May require further processing to 
another format. 


Technology neips to identify and reduce risks during early phases of project 
(Phase A/B) with potential to dramatically reduce overall project costs by 
reducing rework. 

Technology helps identify/reduce mission risks for Phase C/D; Large 
potential cost benefits if used. Provides a screen that limits potential risks 
from passing CDR. 

Technology helps identify technology development or subsystem risks, but 
may or may not influence overall system risk. 


Technology provides new approach for addressing design risk life-cycle or 
part of life-cycle not previously addressed (e.g., mgmt, org. risks) 

Technology either provides new, more effective approach for risk analysis 
or fills missing gap in temporal or breadth of risk analyses (but not both) 
Technology does not address missing gap in design life-cycle. 


End-to-end risk 
integration 


Technology provides synergistic integration with other tools and databases 
fully compatible with emerging design environments (temporal and breadth). 
Risk technology allows interaction with common databases but cannot be 
integrated with other stand-alone applications. 

Technology is stand-alone; focused, narrow; little breadth or temporal range, 
databases are separated with little or no connectivity. Integration difficult. 








All SRRM Technology Areas Are 
Included for the Pilot Study 

1. Risk Methods/Tools (RMT) 

2. Risk Workstation (RWS) 

3. Mishap/Anomaly Database (MAIS) 

4. Model-Based Hazard Analysis (MBHA) 

5. System Complex Research (SCR) 

6. Risk Characterization/Visualization (RCV) 

7. Risk-Based Design (RBDO) 

8. Data Mining Research (DMR) 

9. Investigation Methods/Tools (IMT) 


Typical SRRM Technology Area Data* 


Technology 

Level 

Metric 

Unit 

Polarity 

SOA 

Low 

ML High 

$M 



How performance is 
measured 

What unit 
performance 
is measured 
in 

+ = Better if 
performance 
is higher 
- = Better if 
performance 
is lower 

Current 
state-of-the- 
art for 
similar 
technologies 

Technologist’s estimate 
of low, most likely, and 
high values of what will 
be provided to the 
mission 

How much the 
technologist 
needs to 
achieve TRL 6 
in $M 

ECS 

1 









SRRM 

2 









RISK Methods & 
Tools 

4 

Accessibility of Historical 
Risk Event Data 

0-10 

+ 

4 

7 

8 

9 

2 



Potential to Understand and 
Reduce Design Risks and 
Optimize Resources to Retire 
Risk 

0-10 

+ 

1 

7 

8 

9 




Risk Model Enhancement 
(Potential for Better Model 
Credibility) 

0-10 

+ 

2 

9 

10 

10 




End-to-end Risk Integration 
for Breadth of Domain 

0-10 

+ 

2 

8 

9 

10 




Extent of Needs Covered 

0-1 

+ 

0.5 

0.7 

0.8 

0. 

9 



*SRRM data cast in same format used for all other technologies (shown in slide 14) 
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Mission-Technology Complexity Map 


Electric Propulsion 
Chemical Propulsion 
Radio-Thermal-Electric Power 
Reaction Control 
Multifunction Structures 
Deployable Structures 
Fuel Storage & Control 
Environmental control 

Thermal Control 
Autonomous Nav & Docking 
Temperature Sensors 
Pressure Sensors 
Position Sensors 

High Temperature Electronics for Sensors (CMOS) 
Multi-Sensor Integration 
Actuators Operating at High-Temperatures 
High-Temperature Electronics for Actuators (CMOS) 
Permanent Magnets (Cobalt-Samarium) 

High Temperature Batteries (Primary) 

High Temperature Batteries (Re-Chargeable) 

Phase Change Material Thermal Storage 

Thermal Insulation 

Thermal Switches 

Heat Pipes 

Active Refrigeration 

Pressure Vessel 

Smart Surface Coatings 

Sulfuric Atmosphere Protection 

Robotic In-Space Assembly 

Robotic In-Space Inspection 

Robotic In-Space Maintenance 

Surface Mobility 

Aenal Mobility 

Subsurface Ice Mobility 

Micro-g/Cryovac Mobility 

Manipulation 

Dnlling 

Sampling 

Investigating Methods/Tools 
Data Mining Research 
Risk Based Design 
Risk Charactenzation/Vi sualizalion 


|=1-2 technologies 
=3-4 technologies 
|=5 or more technologies 
=missing data 
j =possible tech need 




Analysis Options Used to Get Typical Results 

in Slides 25-30 


Analysis Options Used 

Other Options Available 

Uniform science-return value for all 
missions 

Can assign non-uniform science return 
value (user prescribed) 

Uniform value for all technologies at the 
same hierarchical level; “democratic” 
hierarchy 

Can prescribe general technology 
organizations; based for example on mission 
and system decomposition 

Technology correlations and co- 
dependencies set to zero 

Can explicitly include correlation & co- 
dependency parameters when available 

Risk estimates based only on performance 
uncertainty 

Can include cost, schedule and other risk 
factors 

Identical development time (-10 yrs) for all 
technologies 

Can vary technology development time as a 
model parameter 

TRL data not included in technology 
projections 

Can analyze TRL data within existing 
analysis framework 
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• MSL 


Estimated Impact of Technology Budgets 



- MSR 

Mars Astrobiology Field Lab 
Titan Explorer 
XVenus Sample return 

• Europa 

+ Comet Sample Return 
-Lunar sample return 

- Space Station maintenance 
Large Observatory Platform 
Gateway 

Space Solar Power 
OASIS 

• HPM 

• CPM 
SEP 

- CTV 

- Venus Surface mission 
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Concluding Remarks 

• Study Results to Date (January-March, 2004) 

- Initial data base for 13 missions and 167 technology performance 
parameters in 23 technical areas, representing Code T,S,M,Y 
enterprises 

- Rapidly prototyped analysis capability to evaluate impact of 
technological investment on science and exploration return 

• Work Remaining (April-December, 2004) 

- Expand data base to include more enabling missions and 
technologies (e.g. modular distributed structures, etc.) 

- Conduct more in-depth analysis of the representation and fidelity 
of the existing data set, and a more detailed treatment of the 
consistency and integration across program elements 
Calibrate data base and analysis with extensive WHAT-IF 
computational 
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Appendix B: Records of Group Discussions 


• Questions for Working Groups 


Questions for Working Groups 

1 . In prioritizing technology development for missions, 
how should the relative values of the missions be 
assessed and quantified? (one measure of relative worth 
is the value that NASA is willing to pay for these missions, 
but there may be better figures of merit in terms of 
information returned? How do you compare value of 
technology supporting Station to that supporting Mission to 
Planet Earth? Within Space Science, how would the value 
technology contribution to a Mars sample Return be 
compared to that which supports a Europa mission? 


Possible Answers 

1 . Should mission (= flight project) value be assessed at all? 

■ Value is always assigned: current processes do this in a non-traceable, non- 
auditabie way. 

■ Has to be done, so that we can improve on today’s process. 

■ Difference between valuation theory and results vs. x decision-makers final 
assessment. 

2. Who should do it? Can it be done? (problem of different 
stakeholders) 

• Code B assesses relative value of missions (they allocate resources to 
Enterprises): 

— Ex: 18 theme areas and 3 mission areas: high, medium, low ranking. 

• Enterprises: Code B apportions resources as a block to Enterprises 

— Enterprises prioritize missions 

• Executive Council, Joint Strategic Assessment Committee. 

— Science Groups/Project Managers: Prioritize missions, 
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Possible Answers 

1 . Should mission (= flight project) value be assessed at all? 

■ Value is always assigned: current processes do this in a non-traceable, non- 
auditable way. 

■ Has to be done, so that we can improve on today’s process. 

■ Difference between valuation theory and results vs. x decision-makers final 
assessment. 

2. Who should do it? Can it be done? (problem of different 
stakeholders) 

• Code B assesses relative value of missions (they allocate resources to 
Enterprises): 

— Ex: 18 theme areas and 3 mission areas: high, medium, low ranking. 

• Enterprises: Code B apportions resources as a block to Enterprises 

- Enterprises prioritize missions 

• Executive Council, Joint Strategic Assessment Committee. 

— Science Groups/Project Managers: Prioritize missions. 


Questions for Working Groups 

2. There are many architectures that might purport to 
enable a mission concept, but at the early formulation 
stage, how might we best select among them, and 
perform a functional decomposition to determine 
quantified capability requirements? 

• How do we get functional requirements at pre-phase A 
stage? 

• Are there better ways to define the science/ops interface 
than fitting the boxes a posteriori? 
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Possible Answers 

1. Is it possible to obtain mission capability requirements at this 
stage? 

• Science mission concepts are typically more mature/have clearer objectives than human 
missions. 

• Assume new undefined missions requirements can be drawn from a spectrum of past 
missions 

• Assume that the requirements evolve from the technological state of the art (technology 
push) and iterate 

2. Advantages and disadvantages of requirements 

• “Requirements” are not ironclad, have to be negotiable. Requirements have to be coupled 
with affordability and serve as a basis for negotiation. 

• Requirements should be expressed quantitatively. Requirements are different from specs. 
Quantification of requirements brings problems, but also allows one to know when one is 
done. 


Possible Answers 

3. Defining mission concepts involves working in a very large trade space. How do 
you search it? 

• Search trade space hierarchically, keeping the number of options low at each level. 

• Delay decisions on final designs: NASA tends to dive into a specific point design too early. A more 
extensive assessment of the trade space, keeping uncertainties and open options, allows a broader, 
more valuable set of technologies to be developed. On the other hand, there are huge costs associated 
with keeping options open. 

4. What technologies should be funded? 

• General technology areas can be extracted from early mission concepts, and these should be funded. 

• Insist that each mission concept study provides one or more functional decompositions (stored in a 
database). Since there is only a limited number of feasible architectures, they can be specified and a 
common set of relevant technologies extracted. Also identify key enabling technologies and perform 
gap analysis. 

• Sustainability is essential, not just affordability. Reusability: define/develop technology building blocks 
that can be “robust" and used across different missions. Avoid cutting off early promising technology 
paths. Temporal impact of technologies has to be taken into account. 
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Questions for Working Groups 

3. How do we systematically acquire credible 
information on technology development 
(cost/performance estimates and associated 
uncertainty, temporal and functional 
correlations etc.) which might seek to satisfy 
capability requirements. 


Possible Answers 

> Add extra fields as part of the Technology Inventory 
collection process 

> Augment the existing CRAI activity with independent 
review. 

> Examine the limits of what might be feasible; 
remember to strive for plausibility not perfect accuracy 

> Have NASA pay for this data acquisition as part of 
system studies 

> Develop models based on historical data 
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Questions for Working Groups 

4. What is the best methodology to perform 
technical risk assessments and mitigations; is 
the evaluation of these fundamentally different 
from the discipline product technologies (e.g. 
sensing, manipulation, mobility etc.). 


Possible Answers 

> Based on experience, assess the objectivity and 
usefulness of quantitatively measuring relative 
reliability gain associated with improved risk 
methodologies 

> Based on mission experience, determine whether new 
risk methodologies are needed. 

> Risk technologies Can/Cannot be blended uniformly 
into a prioritization methodology 
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Questions for Working Groups 

5. What are the criteria management needs to take 
and use the results of such a structured 
analysis. 


Possible Answers 

> Need a sense of confidence in the overall mission 
requirements and technological characterization 

> Consistency with the unstated policies from NASA (re: 
value, pull/push, etc.) 

> Timely response 

> Data acquisition process needs to be feasible from the 
viewpoint of overall effort. 
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Questions - Day 1 


Questions-Day 1 

> How do we systematically acquire credible 
information on technology development 
(cost/performance estimates and associated 
uncertainty, temporal and functional 
correlations, etc.) which might seek to 
satisfy capability requirements? 

■ Credible: presentation would be plausible as seen 
by an independent review team 


A. How do we systematically acquire credible information... 

> Are the data models and assumptions traceable and transparent? 

■ Workshop for credibility review 

■ Peer reviews/third party teams 

* Explicit inclusion of uncertainty for high risk or non-legacy items 

■ Matching capability requirements to technology tasks 

> Sustainable process? (i.e., are iterations easier than first 
bounce?) 

■ POP process as a vehicle for data generation - incentives for proper behavior 

■ Continuing reevaluating process 

■ Quarterly review with researchers and mission experts 

> Are all valid viewpoints considered? 

> Do you have an estimate of the robustness of the conclusions? 

> Do independent review teams have recommendations? 
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B. What is the best methodology... 

> How can the representation and assessment of risk 
estimation/software technologies be made 
consistent with those of the discipline product 
technologies (e.g., sensing, manipulation, mobility, 
etc.)? 

■ Important to have researchers state what kind of metric 
they hope to impact; missions should provide goals 

■ Look at cost impacts as well as performance impacts 

■ Combine software and hardware at a capability level as 
opposed to a discipline level 

■ State-of-the-art can be characterized, but perhaps the 
whole ‘ecosystem’ of software should be looked at, not, for 

instance, an algorithm. .. 


C. What are the criteria that management needs... 

> What are the criteria that management needs to take and 
use the results of such a structured analysis? 

■ Analysis has to support/defend the eventual decision to OMB and GAO 
and others 

• Traceable, transparent, understandable, presented in a concise way 

• Make issues explicit, identify problem areas 

■ Analysis has to address what the decision maker cares about -- 
metrics, alternatives, etc. 

■ Context is decision support 

• Cast as risk vs. cost; benefit vs. cost; 

• Provide options - not point solutions 

- Preferably with recommendations and justifications (not just negatives and 
consequences); span decision space 

• Digestible products tuned to appropriate level 
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Questions - Day 2 


Questions-Day 2 

> In prioritizing technology development for 
missions, how should the relative values of the 
mission be assessed or quantified? 

> There are many architectures that might purport to 
enable a mission, but at the early formulation 
stage, how might we best select among them, and 
perform a functional decomposition to determine 
quantified capability requirements? 


How should the relative values of the missions be assessed? 

> In prioritizing technology development for missions, how should 
the relative values of the missions be assessed or quantified? 

■ “All missions are equal; some are more equal than others” 

■ Aim for functional objectives 

■ Missions fit under some exploration obj. Need a way to handle 
different msn approaches 

■ Start with unity 

i Then apply dollar values to missions 

■ Mission value parametric and subject to multiple interpretation 

• Position in launch queue 

• Normalize all to one 

• Alternative assumptions... etc. 

% Point is they can be varied 




Architecture selection; functional decomposition 

> There are many architectures that might purport to 
enable a mission, but at the early formulation 
stage, how might we best select among them, and 
perform a functional decomposition to determine 
quantified capability requirements? 

> Missions map to technologies that map to metrics 

> Architectures are snapshots of different 
technology metric sets 

> Compare the architectures indirectly by evaluating 
their technology portfolios and costs. 


Architecture selection; functional decomposition 

> Functional decomposition derived from mapping of 
mission capability requirements to technology metrics. 

1 . Obtain capability requirements from mission(s) to level available 

2. Get technology gaps from mission 

3. Map relevant technologies to capability requirements 

4. Derive performance metrics for technologies 

5. Evaluate fulfillment of requirements by performance (simulation, 
modeling, figures of merit) 

6. Weight by parametric mission values; sensitivity analysis 

> Don’t over-weigh optimizations but consider level of 
precision; reserve some fraction for visionaries and 
spontaneous discoveries 

> Consider approaches from other sectors (gov’t., non- 
NASA, public, etc.) 
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